PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation

One of the most efficient methods for model compression is hint distillation, where student injected with information (hints) from several different layers teacher model. Although selection points can drastically alter performance, conventional distillation approaches overlook this fact and use same as in early studies. Therefore, we propose a clustering based methodology, are clustered respect to metrics cluster centers used points. Our method applicable any network, once it applied on chosen network. The proposed approach validated CIFAR-100 ImageNet datasets, using various teacher-student pairs numerous methods. results show that selected by our algorithm superior performance compared state-of-the-art knowledge algorithms models datasets.